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Executive  Summary 

The  711  Human  Performance  Wing  Continuous  Learning  Branch  (71 1HPW/RHAS)  has  been 
working  with  Air  Combat  Command/  (ACC/A2RT)  for  the  past  5  years  in  the  Intelligence, 
Surveillance,  and  Reconnaissance  (ISR)  training  research  domain.  During  an  information 
exchange  meeting  in  January  2008  at  Langley  AFB,  ACC/A2RT  requested  help  to  find  empirical 
evidence  that  links  critical  thinking  to  intelligence  analysis.  Specifically,  is  it  possible  to  teach 
critical  thinking  skills  to  intelligence  analysts  to  improve  their  analytical  skills,  and  what  would  a 
critical  thinking  curriculum  consist  of?  This  report  describes  initial  review  of  published 
literature  on  critical  thinking  skills  and  training  to  enhance  skills  as  they  relate  to  improving 
performance  of  intelligence  analysts. 

For  this  effort,  with  the  support  of  two  Air  Force  Academy  cadets,  we  conducted  a  review  of 
published  literature  within  the  psychology,  education,  and  intelligence  domain  related  to  critical 
thinking  skills.  From  this  initial  review,  I  produced  a  more  comprehensive  report  detailing:  1) 
what  we  know,  2)  what  we  do  not,  3)  what  methods  have  been  used  to  train/develop/promote 
what  we  defined  as  critical  thinking  as  it  relates  to  intelligence  analysis,  and  4)  what  are  some  of 
the  challenges  and  issues. 

Currently  few  have  thoroughly  investigated  and  written  about  the  linkage  between  critical 
thinking  and  intelligence  analysis.  We  can,  however,  draw  upon  the  research  in  critical  thinking 
in  the  educational  and  psychology  domain  to  lead  us  forward.  Within  these  domains,  there  are  a 
variety  of  definitions  of  critical  thinking  and  its  components.  There  are  also  a  plethora  of 
curriculums  developed  that  claim  to  improve  critical  thinking.  We  must  identify  the  critical 
thinking  skills  as  it  relates  to  intelligence  analysis  before  we  can  identify  a  training  curriculum. 

A  second  issue  exists  as  there  is  currently  no  true  assessment  for  intelligence  analysis.  Much 
more  research  is  needed  to  provide  an  answer  to  how  critical  thinking  may  improve  intelligence 
analysis. 
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Behavioral  and  Psychosocial  Considerations  in  Intelligence  Analysis 


Background 

Two  Air  Force  academy  cadets,  C1C  Mark  Bailie  and  Trent  Atwood,  worked  at  ARFL  in  Mesa 
for  a  summer  research  internship.  During  their  stay  in  Mesa,  they  produced  an  initial  annotated 
bibliography  that  kick  started  the  research  (appendix  A).  In  the  following  months,  I  continued  to 
broaden  the  literature  search. 

After  the  initial  review  in  current  literature,  we  found  that  few  have  investigated  the  link  between 
critical  thinking  and  intelligence  analysis.  While  there  has  been  work  trying  to  link  the  two, 
there  has  not  been  any  empirical  evidence  showing  the  correlation.  Many  authors  speculate  that 
the  link  does  exist  but  no  definitive  correlation  exists  currently.  In  order  for  that  to  happen,  two 
issues  must  be  resolved:  a)  criteria  matrix  for  scoring  intelligence  analysis  and  b)  tools  or 
methodologies  to  quantitatively  measure  that.  At  the  present  neither  of  these  issues  has  been 
solved.  Within  the  limited  scope  of  this  effort,  it  may  be  unrealistic  to  establish  a  definitive  link 
between  critical  thinking  skills  and  intelligence  analysis.  Rather  than  trying  to  solve  these  issues 
in  this  effort,  we  began  to  identify  within  the  instructional  design  and  learning  theory  literature 
on  how  novices  learn.  Furthermore,  we  propose  a  possibility  of  using  digital  games  to  teach 
critical  thinking  in  regards  to  intelligence  analysis  and  mapping  the  rationales  back  to  established 
principles  of  learning  theories. 

Methods 

In  a  response  to  ACC/A2RT’s  request  for  help,  we  began  a  broad  literature  search  on  relevant 
literature  in  the  educational,  psychology,  and  operational  intelligence  domain.  I  combined 
relevant  findings  in  these  disparate  fields  to  address  some  of  the  issues  on  critical  thinking  as  it 
relates  to  intelligence  analysis. 

Results 

Critical  Thinking 

After  the  initial  literature  search,  I  focused  on  critical  thinking  skills  and  training  strategies 
designed  to  enhance  a  person’s  critical  thinking  skills.  Within  the  education  and  training 
communities,  there  is  a  myriad  of  definitions  for  critical  thinking  and  critical  thinking  skills. 
According  to  Moore  (2007),  critical  thinking  is  defined  as: 

...  a  deliberate  meta-cognitive  and  cognitive  act  whereby  a  person  reflects  on  the  quality 
of  the  reasoning  process  simultaneously  while  reasoning  to  a  conclusion.  The  thinker  has 
two  equally  important  goals:  coming  to  a  solution  and  improving  the  way  she  or  he 
reasons,  (p.8) 

Another  similar  definition  by  Halpern  (2002): 
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. .  .critical  thinking  is  the  use  of  those  cognitive  skills  or  strategies  that  increase  the 
probability  of  a  desirable  outcome.  It  is... thinking  that  is  purposeful,  reasoned,  and  goal 
directed  -  the  kind  of  thinking  involved  in  solving  problems,  formulating  inferences, 
calculating  likelihoods,  and  making  decisions,  when  the  thinker  is  using  skills  that  are 
thoughtful  and  effective  to  the  particular  context  and  type  of  thinking  task.  (p.  37) 

Just  as  there  is  no  universal  definition  for  critical  thinking,  there  is  no  consensus  on  what 
qualifies  as  critical  thinking  skills.  From  February  1988  to  the  end  of  November  1989,  the 
American  Philosophical  Association  put  together  a  panel  of  experts  from  the  Education,  Social 
Sciences,  and  Physical  Sciences  domain  to  participate  in  the  Delphi  project.  The  Delphi  project 
was  an  attempt  to  come  to  some  consensus  statement  regarding  critical  thinking  and  the  ideal 
critical  thinker  and  present  recommendations  for  future  work.  Under  this  project,  Facione  (1990) 
composed  a  consensus  set  of  critical  thinking  cognitive  skills  and  subskills.  The  skills  and 
subskills  are  listed  in  table  1  below. 


Table  1 

Consensus  list  of  CT  Cognitive  Skills  and  Sub  Skills 

Skills 

Subskills 

1)  Interpretation 

Categorization 

Decoding  significance 

Clarifying  meaning 

2)  Analysis 

Examining  ideas 

Identifying  arguments 

Analyzing  arguments 

3)  Evaluation 

Assessing  claims 

Assessing  arguments 

4)  Inference 

Querying  evidence 

Conjecturing  alternatives 

Drawing  conclusions 

5)  Explanation 

Stating  results 

Justifying  procedures 

Presenting  arguments 

6)  Self-regulation 

Self-examination 

self-correction 
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What  is  clear  is  that  there  seem  to  be  some  overlaps.  That  is  that  experts  tend  to  agree  that  there 
are  some  commonalities  amongst  the  varied  understanding  of  critical  thinking.  As  Lipman 
(1988)  suggests  critical  thinking  is  “skilful,  responsible  thinking  that  facilitates  good  judgment 
because  it  (1)  relies  upon  criteria,  (2)  is  self-correcting,  and  (3)  is  sensitive  to  context"  (p.  39). 
Lipman  agrees  that  the  quality  of  thinking  is  more  important  than  the  process  of  thinking;  that  we 
need  to  employ  criteria  upon  which  to  examine  our  thinking  and  facilitate  judgment.  A  second 
component  to  critical  thinking  is  that  thinking  is  self-correcting.  At  every  step  of  thinking,  we 
reflect  on  our  thinking  against  the  criteria  we’ve  established.  And  finally  that  the  criteria 
established  will  be  both  general  and  context  specific. 

In  regards  to  correlation  between  critical  thinking  and  intelligence  analysis,  Heur  (1999)  is  one 
of  the  few  notable  authors  defining  the  issue.  He  suggests  that  errors  occurring  in  analysis  are 
due  to  limitations  of  cognition.  He  cites  that  it  is  cognitive  biases  or  simplified  information 
processing  strategies  that  lead  to  flawed  analysis.  He  defines  these  as:  selectivity  bias, 
availability  bias,  absence  of  evidence  bias,  confirmation  bias,  overconfidence  bias, 
oversensitivity  to  consistency  bias,  and  discredited  evidence  bias.  Definitions  of  each  type  of 
biases  are  in  table  2.  These  biases  have  been  thoroughly  researched  and  demonstrated  in  the 
numerous  psychological  experiments. 


Table  2 

Types  of  Heur’s  biases 

Selectivity  bias 

Information  is  selectively  recalled  as  a  function  of  how  vivid, 
concrete,  and/or  personal  an  event  is 

Availability  bias 

Prediction  of  the  frequency  of  an  event  is  based  on  how  easily  it 
is  recalled 

Absence  of  evidence  bias 

Most  do  not  recognize  and  incorporate  missing  data  into 
judgment  of  abstract  problems 

Confirmation  bias 

Tendency  to  perceive  events  in  such  a  way  as  to  confirm 
existing  beliefs 

Overconfidence  bias 

Overconfidence  in  one's  own  judgment  on  that  they  are  correct 
when  in  fact  most  of  the  time  they  are  wrong 

Oversensitivity  to  consistency 
bias 

Tendency  to  place  too  much  reliance  on  small  samples  or  the 
inability  to  discern  multiple  reports  from  same  source 

Discredited  evidence  bias 

Impressions  tend  to  persist  even  after  the  evidence  that  created 
those  impressions  has  been  fully  discredited. 
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In  order  to  prevent  these  biases  from  occurring,  the  analyst  must  understand  what  they  are  and 
reflect  upon  them  at  each  stage  of  analysis.  In  other  words,  metacognition  or  reflective  thinking 
is  an  important  part  of  good  analysis.  Moore  (2007)  further  maintains  that: 

Critical  thinking  mitigates  error  by  providing  means  to  assess  errors  in  reasoning  as  they 

occur  and  before  they  become  systematic  failures  (p.  81) 

He  asserts  that  self-questioning  must  occur  at  every  analytical  step.  That  is  reflective  thinking 
when  assessing  information,  assessing  evidence,  creating  inferences,  and  in  producing 
intelligence  is  essential. 

Learning  Theories 

To  begin  formulation  of  strategies  for  teaching  critical  thinking  skills,  the  issue  of  domain 
transfer  versus  domain  dependent  must  be  tackled.  There  is  evidence  to  suggest  that  both  may 
be  occurring.  McMurray  and  Thompson  (1989)  and  Anderson  and  Soden  (2001)  found  that 
critical  thinking  skills  learned  in  one  domain  did  not  easily  and  readily  transfer  to  a  novel 
domain.  However,  Lehman  and  Nisbett  (1990),  Kosonen  &  Winne  (1995),  and  Nisbett  (1993) 
found  that  skills  obtained  in  one  domain  did  indeed  transfer  to  another  domain.  Halpern  (1998) 
proposed  a  model  of  teaching  critical  thinking  skills  to  facilitate  the  transfer  across  domains. 

The  model  consists  of  four  parts:  1)  a  dispositional  or  attitudinal  component,  2)  instruction  in 
and  practice  with  critical  thinking  skills,  3)  structure-training  activities  designed  to  facilitate 
transfer  across  contexts,  and  4)  a  metacognitive  component  used  to  direct  and  assess  thinking. 

For  the  purpose  of  this  review,  we  will  take  the  stance  that  critical  thinking  skills  can  be  transfer 
across  domains.  That  is  teaching  general  critical  thinking  skills  will  lead  to  improvement  in 
intelligence  analysis. 

Drawing  from  the  education  domain  and  learning  theory,  there  also  seems  to  be  no  consensus  as 
to  what  approach  is  best  for  learning.  Programs,  strategies,  and  training  programs  are  important 
but  for  it  to  be  effective  it  is  contingent  on  a  variety  of  other  factors.  One  such  factor  is  the 
student’s  current  ability.  Clark  and  Wittrock  (2000,  as  cited  in  Goodwin  2006)  found  a  big 
interaction  between  aptitude  and  training.  He  conceptualized  training  to  be  of  four  types  that 
range  on  a  continuum  between  external  to  internal  factors  to  the  learner.  The  four  types  of 
training  are:  receptive  (teaching  by  telling),  behavioral  (teaching  by  demonstration  and 
feedback),  guided  discovery  (teaching  by  problem  solving),  and  exploratory  (teaching  by 
exploration).  Research  suggests  while  guided  discovery  and  exploratory  training  worked  well 
for  individuals  with  high  aptitude  and  motivation,  it  did  not  work  well  for  novices  with  low 
aptitude.  The  important  implication  here  is  to  tailor  the  training  approach  to  the  student’s  level 
of  ability.  Additionally,  drawing  from  cognitive  load  theory  novices  initially  learn  better  in  well 
structured  learning  environment  (Jonassen,  1997).  That  is  learning  using  worked  out  examples  is 
effective  for  novices  as  noted  by  the  worked  example  effect  (Clark,  Nguyen,  and  Sweller,  2006). 
Using  worked  examples,  teachers  can  use  scaffolding  by  modeling  the  desired  learning  strategy 
or  task.  However,  as  a  learner  becomes  more  knowledgeable  there  is  a  gradual  decline  in  the 
effectiveness  in  using  well  structured  learning  environments  such  as  demonstrated  by  the 
expertise  reversal  effect  (Kalyuga,  Ayres,  Chandler,  and  Sweller,  2003).  The  consensus  is  that  a 
continuum  of  well  structured  to  ill- structured  learning  environments  be  used  as  learners  become 
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more  knowledge  in  the  domain.  Problem  solving  scenarios  are  more  effective  during  the  later 
stages  of  learning;  where  the  learner  has  to  define  the  problem.  Additionally,  in  the  beginning 
stages  of  learning  collaborative  environments  have  been  shown  to  be  effective.  Remedios, 

Clarke  and  Hawthorne  (2008,  p.2)  suggests  that  it  is  “the  opportunity  to  share  a  large  workload, 
learn  from  multiple  perspectives,  distribute  the  cognitive  load,  negotiate  shared  understanding, 
develop  social  skills,  and  function  as  a  content  expert  for  a  group  peer”  that  has  been  deemed 
advantageous  through  collaborative  learning.  Because  of  this,  a  group  working  together  has  a 
greater  potential  for  deeper  understanding  than  an  individual  working  alone. 

Correlation 

Currently  there  is  no  empirical  evidence  suggesting  that  there  is  a  correlation  between  critical 
thinking  and  improved  intelligence  analysis  because  few  have  investigated  the  correlation.  More 
importantly,  evidence  currently  only  shows  a  weak  correlation  between  critical  thinking  training 
and  improvements  in  thinking.  Part  of  the  problem  is  that  there  exist  a  plethora  of  critical 
thinking  assessment  techniques  and  tests.  Each  has  its  advantages  and  disadvantages  but  there 
seems  to  be  no  consensus  on  which  is  more  effective.  Each  seems  to  measure  a  different  aspect 
of  critical  thinking.  The  Mental  Measurement  Yearbook  from  the  Buros  Institute  of  Mental 
Measurements  has  a  comprehensive  list  of  reviews  of  critical  thinking  assessment  tools  and 
methodologies. 

I  found  one  paper  that  demonstrates  an  improvement  in  critical  thinking  after  a  program  of 
training.  Twardy  (2004)  found  that  argument  maps  statistically  improved  critical  thinking  on  pre 
and  post-test  as  measured  by  the  California  Critical  Thinking  Skills  Test  (CCTST).  Argument 
maps  are  the  visual  representation  of  the  structure  of  an  argument.  The  rationale  behind 
argument  maps  is  that  learners  can  visually  identify  the  errors  in  the  line  of  reasoning. 

Current  Intelligence  Curriculums 

As  of  2003,  the  CIA  Sherman  Ken  School  for  Intelligence  analysts  included  critical  thinking 
development  as  part  of  its  curriculum.  The  NSA  initiated  a  40  hour  program  to  enhance  the 
critical  thinking  skills  of  analysts.  In  2005,  DIA  started  a  similar  program  to  develop  critical 
thinking  skills  in  their  analysts.  The  Regional  Joint  Intelligence  Training  Facility  in  Molesworth, 
UK  has  a  course  on  Critical  Thinking  and  Structured  Analysis.  These  are  only  a  small  sample  of 
current  programs  available.  It  is  obvious  that  there  is  an  invested  interest  in  promoting  critical 
thinking  in  the  intelligence  community.  The  current  curriculums  seem  to  focus  more  intelligence 
analysis  techniques  than  solely  developing  critical  thinking.  These  intelligence  analysis 
methodologies  include  techniques  such  as  key  assumption  checks,  analysis  of  competing 
hypothesis,  alternative  future  analysis,  red  team  analysis,  team  A/team  B  analysis,  what  if 
analysis,  etc.  Most  of  these  techniques  incorporate  elements  of  critical  thinking  consistent  with 
many  of  the  components  as  discussed  earlier  such  as  a  metacogntive  or  reflective  component,  a 
dispositional/attitudinal  component,  and  criterion  based  thinking. 

One  interesting  approach  in  training  intelligence  analysts  is  through  the  use  of  computer  based 
games.  The  DIA  had  a  $2.6  million  contract  to  develop  three  PC-based  games  for  quickly 
training  new  analysts.  All  three  games  put  the  learner  in  the  position  of  a  young  DIA  analyst. 
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Each  game  plays  out  in  90  minutes  to  three  hours  with  multiple  story  lines.  Game  based  learning 
has  been  a  hot  topic  for  the  DoD  lately.  It  seems  that  research  supports  the  notion  of  using 
games  for  learning.  Becker  (2005)  noted  the  learning  theory  behind  games  by  adopting  Gagne’s 
nine  events  of  instruction  inherently  embedded  in  good  video  games  to  demonstrate  why  games 
are  educational.  These  are:  1)  gaining  attention,  2)  informing  learner  of  objective,  3)  stimulating 
recall  of  prior  learning,  4)  presenting  the  content,  5)  providing  learning  guidance,  6)  eliciting 
performance,  7)  providing  feedback,  8)  assessing  performance,  and  9)  enhancing  retention  and 
transfer. 

Discussion 

While  it  is  apparent  that  there  is  an  increased  investment  by  the  intelligence  community  in 
critical  thinking  curriculum,  there  currently  is  a  lack  of  empirical  evidence  correlating 
improvement  in  analysis  through  critical  thinking  training.  To  resolve  this,  assessment 
methodologies  for  intelligence  analysis  must  be  thoroughly  developed.  Only  then  will 
researchers  be  able  to  objectively  measure  pre  and  post  test  results.  Furthermore,  for  any  critical 
thinking  curriculum  to  be  effective,  the  assumptions  behind  the  program  must  be  thoroughly 
defined.  That  is  the  skills  and  subskills  of  the  program  should  be  directed  at  promoting 
intelligence  analysis  and  the  teaching  methodology  should  match  the  learner’s  level  of  ability. 

Future  efforts  should  investigate  critical  thinking  in  both  the  medical  and  business  intelligence 
community.  The  problem  structure  in  the  medical  community  and  business  intelligence  is  very 
similar  to  intelligence  analysis.  For  all  three  communities,  they  must  sort  through  voluminous 
and  disparate  information,  analyze  and  piece  the  data  into  a  coherent  picture,  and  finally  develop 
a  course  of  action  to  deal  with  the  conclusion.  For  intelligence  analyst  they  have  to  find 
information  related  to  the  problem,  analyze  the  information,  and  produce  actionable  intelligence. 
For  the  medic  personnel,  they  have  to  find  the  symptoms  displayed  in  a  patient,  analyze  the 
information  to  come  to  a  diagnosis,  and  develop  a  course  of  treatment.  And  for  business 
intelligence,  an  analyst  must  sort  and  vet  through  considerable  amount  of  information  such  as 
customer  behavior  or  competitor  strategies,  produce  actionable  intelligence,  and  develop 
business  strategies  to  dominate  the  market.  For  all  these  communities  this  is  a  dynamic  process. 
That  is  at  any  point  during  the  process,  new  information  may  be  found  and  the  analysis  must  be 
reassess  or  reevaluate  which  may  or  may  not  change  the  course  of  action  needed  to  deal  with  the 
situation.  Furthermore,  there  is  another  commonality  that  a  fair  amount  of  deceit  is  purposefully 
used  by  the  enemies  to  mislead  the  analyst. 

By  exploring  how  the  medical  and  business  intelligence  community  defines  critical  thinking  and 
how  they  improve  their  analysis,  we  may  find  overlaps  that  are  useful  in  promoting 
improvements  in  intelligence  analysis. 
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Appendix  A 


Annotated  Bibliography 
Cadets  Mark  Bailie  and  Trent  Atwood 

Brightman,  Harvey  J.  GSU  Master  Teacher  Program:  on  Learning  Styles.  Georgia  State 
University.  2007.  25  May  2008  <http://www2.gsu.edu/~dschjb/wwwmbti.html>. 

-  Understanding  how  people  best  learn  is  important  in  the  teaching  process.  This  article 
attempts  to  explain  how  people  learn  and  uses  the  information  on  learning  to  explain  the 
best  way  to  teach.  Through  the  discussion  of  different  types  of  students  along  with 
different  teaching  methods  the  reader  becomes  familiar  with  different  ways  to 
successfully  teach. 

Burbach,  Mark,  Gina  S.  Matkin,  and  Susan  M.  Fritz.  "Teaching  Critical  Thinking  in  an 

Introductory  Leadership  Course  Utilizing  Active  Learning  Strategies:  a  Confirmatory 
Study."  College  Student  Journal  38  (2004):  482-493.  MasterFILE  Premier.  EBSCO.  4 
June  2008. 

-  This  study  was  conducted  to  determine  whether  an  introductory  level  college  leadership 
course  that  encouraged  active  learning  increased  critical  thinking  skills.  A  pre-  and  post¬ 
assessment  of  critical  thinking  skills  was  conducted  using  the  Watson-Glaser  Critical 
Thinking  Appraisal.  Significant  increases  were  found  in  the  Deduction  and  Interpretation 
subtests,  and  total  Critical  Thinking.  Student  engagement  in  active  learning  techniques 
within  the  context  of  studying  interpersonal  skills  for  leadership  appeared  to  increase 
critical  thinking.  (Abstract) 

Clark,  Donald.  Learning  Domains  or  Bloom's  Taxonomy.  3  June  2008 
<http://www.nwlink.com/~donclark/hrd/bloom.html>. 

-  Here  the  types  of  learning  are  broken  down  into  three  groups.  Within  each  group  they 
are  further  broken  down  and  given  examples.  This  article  allows  for  a  better 
comprehension  of  the  types  of  learning  to  appeal  to  as  well  as  to  develop. 


Hatcher,  Donald.  On  Assessing  and  Comparing  Critical  Thinking  Programs:  a  Response  to 
Hitchcock.  Baker  University. 
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-  The  author  analyzes  a  report  done  on  different  methods  of  developing  critical  thinking 
and  analyzing  it.  His  work  focuses  mostly  on  comparing  and  critiquing  Hitchcock’s 
analysis  on  the  same  problem.  Through  this  critique  he  manages  to  give  insight  into  the 
different  ways  of  developing  critical  thinking  skills  through  courses.  The  article  is 
finished  with  comparative  statistics  on  test  scores. 

Heuer,  Richards  J.  United  States.  Central  Intelligence  Agency.  Psychology  of  Intelligence 

Analysis.  1999. 

-  A  comprehensive  review  of  cognitive  literature  in  search  of  how  people  comprehend 
information  and  make  decisions  based  on  ambiguity.  Touches  on  a  number  of  topics 
related  to  cognition,  metacognition,  as  well  as  intelligence  analysis.  The  reader  is  also 
informed  of  different  techniques  for  thinking  as  well  as  bias’  to  be  aware  of.  Overall  this 
report  seeks  to  prepare  analysts  for  advanced  critical  thinking  skills  and  offers  a  method 
to  do  so. 

Hopf-Weichel,  R.,  J.  R.  Thompson,  and  R.  E.  Geiselman.  The  Cognitive  Basis  of  Intelligence 
Analysis.  U.S.  Army  Research  Institute.  United  States  Army,  1984. 

-  This  report  summarizes  the  background  research  that  led  to  development  of  the 
"Strategic  Intelligence  Analysis  Handbook”  which  was  also  developed  under  this 
contract.  The  goal  of  the  research  was  to  develop  a  framework  for  understanding  human 
processes  in  intelligence  analysis  to  be  used  in  the  development  or  evaluation  of  training 
procedures,  doctrine,  and  system  requirements  for  automated  support  to  analysts. 
(Abstract) 

Jones,  Anna.  Teaching  Critical  Thinking:  an  Investigation  of  a  Task  in  Introductory 

Macroeconomics .  University  of  Melbourne.  Higher  Education  Research  &  Development. 
4  June  2008 

<http://search.ebscohost.com/login. aspx?direct=true&db=aph&AN=13310123&site=eho 
st-live>. 

-  This  paper  is  an  investigation  of  understandings  of  critical  thinking  from  two  teaching 
perspectives:  academic  staff  and  tutors.  It  explores  critical  thinking  as  situated  within  an 
assessment  task  in  introductory  macroeconomics.  This  study  found  that  while  the  two 
academic  staff  conceptualized  critical  thinking  as  a  set  of  concrete  cognitive  skills,  the 
tutors  challenged  this  notion. 
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Katter,  Robert  V.,  Christine  A.  Montgomery,  and  John  R.  Thompson.  Human  Processes  in 
Intelligence  Analysis.  Army  Research  Institute.  United  States  Army,  1980. 

-  This  report  provides  an  overview  of  the  results  of  a  study  entitled  “Investigation  of 
Methodologies  and  Techniques  for  Intelligence  Analysis.  The  goal  of  this  study  was  to 
develop  a  framework  for  understanding  human  processes  in  intelligence  analysis. 

An  understanding  of  these  processes  will  be  useful  during  the  development  or  evaluation 
of  training  procedures,  doctrine,  and  system  requirements  for  automated  support  to 
analysts.  (Abstract) 

Moore,  David  T.  Critical  Thinking  and  Intelligence  Analysis.  National  Defense  Intelligence 
College.  Washington  DC:  National  Security  Agency,  2007. 

-  Discusses  the  relationship  between  critical  thinking  and  Intelligence  Analysis.  Through 
the  observance  of  past  examples  in  history  and  a  thorough  understanding  and  discussion 
of  the  topic,  the  authors  discuss  how  to  employ  critical  thinking  and  how  to  teach  it  in 
analysts. 

Reiber,  Steven,  and  Neil  Thomason.  "Creation  of  a  National  Institute  for  Analytic  Methods." 

CIA.  15  Apr.  2007.  CIA.  4  June  2008  <https://www.cia.gov/library/center-for-the-study- 
of-intelligence/csi-publications/csi- 

studies/studies/vol49no4/Analytic_Methods_7.htm#_ftnl9>. 

-  While  much  has  been  written  about  how  to  improve  intelligence  analysis,  this  article 
will  show  how  to  improve  the  process  of  improving  analysis.  The  key  is  to  conduct 
scientific  research  to  determine  what  works  and  what  does  not,  and  then  to  ensure  that  the 
Intelligence  Community  uses  the  results  of  this  research.  (Author’s  Introduction). 

Sormunen,  Carolee,  and  Marilyn  Chalupa.  "Critical  Thinking  Skills  Research:  Developing 

Evaluation  Techniques."  Journal  of  Education  for  Business  69  (1994):  172-178. 

EBSCOhost.  4  June  2008 

<http://search.ebscohost.com/login. aspx?direct=true&db=aph&AN=9403227533&site=e 
host-live>. 

-  Discusses  the  options  when  it  comes  to  evaluating  critical  thinking  skills.  And 
emphasizes  what  is  needed  in  the  realm  of  critical  thinking  education  and  evaluation. 
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Yanchar,  Stephen  C.,  and  Brent  D.  Slife.  "Teaching  Critical  Thinking  by  Examining 

Assumptions."  Teaching  of  Psychology  31  (2004):  85-90.  PsycINFO.  EBSCO.  4  June 
2008. 

-  We  describe  how  instructors  can  integrate  the  critical  thinking  skill  of  examining 
theoretical  assumptions  (e.g.,  determinism  and  materialism)  and  implications  into 
psychology  courses.  In  this  instructional  approach,  students  formulate  questions  that  help 
them  identify  assumptions  and  implications,  use  those  questions  to  identify  and  examine 
the  assumptions  and  implications  of  theories  being  studied,  and  develop  defensible 
positions  on  the  tenability  of  various  theoretical  assumptions.  (Abstract) 
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